-
Notifications
You must be signed in to change notification settings - Fork 850
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve detection of resource attributes on ECS #4574
Improve detection of resource attributes on ECS #4574
Conversation
e9ebf1e
to
46fbf18
Compare
Codecov ReportBase: 90.04% // Head: 90.60% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #4574 +/- ##
============================================
+ Coverage 90.04% 90.60% +0.56%
+ Complexity 5057 4906 -151
============================================
Files 582 567 -15
Lines 15580 14728 -852
Branches 1495 1414 -81
============================================
- Hits 14029 13345 -684
+ Misses 1096 959 -137
+ Partials 455 424 -31
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! This generally looks good, @willarmiros can you also give it a check?
|
||
AttributesBuilder attrBuilders = Attributes.builder(); | ||
static void parseUrl(SimpleHttpClient httpClient, String url, AttributesBuilder attrBuilders) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
static void parseUrl(SimpleHttpClient httpClient, String url, AttributesBuilder attrBuilders) { | |
static void fetchMetadata(SimpleHttpClient httpClient, String url, AttributesBuilder attrBuilders) { |
This improves the detection of resource attributes on ECS by fetching ECS metadata from `ECS_CONTAINER_METADATA_URI` or `ECS_CONTAINER_METADATA_URI_V4`. Previously only `CONTAINER_NAME` and `CONTAINER_ID` id were set. Now we set: - CONTAINER_ID - CONTAINER_NAME - AWS_ECS_CONTAINER_ARN - CONTAINER_IMAGE_NAME - CONTAINER_IMAGE_TAG - aws.ecs.container.image.id - AWS_LOG_GROUP_ARNS - AWS_LOG_GROUP_NAMES - AWS_LOG_STREAM_NAMES - AWS_ECS_TASK_ARN - AWS_ECS_TASK_FAMILY - AWS_ECS_TASK_REVISION Especially AWS_LOG_GROUP_ARNS is important so that connection of traces to logs works OOTB on X-Ray.
46fbf18
to
d0363a2
Compare
Thanks so much for this improvement! I will have someone from AWS take a look at this shortly |
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
Bump: Please don't close this PR automatically. |
@willarmiros any update here? |
Apologies for the delay here, we will be having someone review this shortly! |
@felixscheinost i missed out on this PR and worked on #4670. Your PR is effectively a superset of mine, with the exception of |
@mmanciop Thanks for rebasing your changes on my PR! Regarding the change in the log group ARN. That dosn't seem right to me. Have you tested this on AWS? I just changed this line on my branch, to test this change in isolation, and deployed to our test evironment.
With this change in place I can no longer see the logs for a trace in CloudWatch. If I view a trace, for which I previously could view logs, and click on "Resources" for "Cloudwatch logs" I see the following:
I also tried adding another separting
But that is also wrong as my log group is just called |
I compared the ARNs with the output of |
@mmanciop When I do Only when the log group name contains |
I tried it on ECS, but this bears more validation. Brb :-) |
So, what I see in the ECS console when deploying a simple CDK + Amazon ECS project is this:
The log-group is called The AWS CLI reports:
So, yes, we should not add programmatically |
@felixscheinost I pushed a commit to remove the spurious |
Yeah, I think so. It seems like it does work without it but both the CLI and aws.amazon.com display log groups with that trailing Okay, I will manually test the other changes from your commit after the weekend and if everything looks fine (which I assume it will), pull them into this PR. Is it okay for you if I squash your two commits into one and push it on top of this branch? |
Yup, I think in the end it is best to squash the entire PR on merge anyhow |
Hi @mmanciop, unfortunately even the trailing I cherry picked all your changes, deployed and tried showing the logs for a trace but didn't see any. As sometimes the logs take a while to show up I waited and tried again but still no logs. Then I removed the trailing The output for the trace under "Resources" -> "CloudWatch logs" is:
vs this:
So it seems CloudWatch is very peculiar about how the log group ARNs are formatted. Maybe someone from AWS can verify this. |
@felixscheinost how exactly are you using these ARNs? With which tool? Because the issue could be there too. Besides, the spec clearly states with the example that the |
I am using CloudWatch traces on the official AWS web console. |
These are merely "examples". This links also references the official AWS documentation, which states:
I think in the end what works is what's correct. 3rd party tools which parse |
I suppose @willarmiros, @Aneurysm9 should be making the call and, if |
Sorry for being late here: so from my reading there were 2 issues:
I will submit a fix for the bug so that both ARN formats work. The spec should stay as-is because the examples are still valid formats, just our parsing code is wrong. However, for now it would be best if the resource detectors could record the ARNs without the trailing |
assertThat(attributes) | ||
.containsOnly( | ||
entry(ResourceAttributes.CLOUD_PROVIDER, "aws"), | ||
entry(ResourceAttributes.CLOUD_PLATFORM, "aws_ecs"), | ||
entry(ResourceAttributes.CONTAINER_NAME, InetAddress.getLocalHost().getHostName())); | ||
entry(ResourceAttributes.CONTAINER_NAME, "ecs-curltest-24-curl-cca48e8dcadd97805600"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change "24" to "26" for consistency.
{ | ||
"DockerId": "ea32192c8553fbff06c9340478a2ff089b2bb5646fb718b4ee206641c9086d66", | ||
"Name": "curl", | ||
"DockerName": "ecs-curltest-24-curl-cca48e8dcadd97805600", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change "24" to "26" for consistency.
"com.amazonaws.ecs.container-name": "curl", | ||
"com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/8f03e41243824aea923aca126495f665", | ||
"com.amazonaws.ecs.task-definition-family": "curltest", | ||
"com.amazonaws.ecs.task-definition-version": "24" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be "26" to match ecs-task-metadata-v4.json
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!
Just requested a small change to keep the task-definition-version number consistent in the V4 files.
@willarmiros I cannot push to @felixscheinost's branch (I have to push rights for his fork, which he used to open this PR), but I prepared the commit on https://github.com/lumigo-io/opentelemetry-java/tree/pr-for-felix . |
`CONTAINER_NAME` and `AWS_ECS_TASK_REVISION` should match
This commit adds implementations for the `aws.ecs.launchtype` and `aws.logs.stream.arns` attributes, as well as fixing the generation of log group ARNs.
I pushed another commit which removes the trailing |
Both with and without trailing `:*` are valid formats but there is a bug in the OpenTelementry collector which can’t handle the trailing `:*` (for now) (see open-telemetry/opentelemetry-collector-contrib#13702) So remove addition of the trailing `:*` for now.
27edbd8
to
1fc24b4
Compare
@willarmiros when you think this is ready, please approve and we can move forward. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
This improves the detection of resource attributes on ECS by fetching ECS metadata from
ECS_CONTAINER_METADATA_URI
orECS_CONTAINER_METADATA_URI_V4
.Previously only
CONTAINER_NAME
andCONTAINER_ID
id were set.Now we set:
Especially
AWS_LOG_GROUP_ARNS
is important so that connection of traces to logs works OOTB on X-Ray.